Docker: Chain egress proxy through upstream HTTP(S)_PROXY - #2142

Builds on the upstream-chaining work in unclecode#2142, which asks the upstream to
CONNECT to an already-pinned IP so the rebinding guarantee holds. That is the
right default, but it cannot serve an upstream that fronts a network whose
names do not resolve on our side, or one that enforces hostname ACLs and so
refuses CONNECT-to-an-IP — both listed there as known limitations.

CRAWL4AI_UPSTREAM_PROXY_DNS_SUFFIXES names the suffixes an operator wants the
upstream to resolve. Matching hostnames skip resolve_and_pin and are sent to
the upstream verbatim; the upstream then owns where the connection lands.

Two layers had to honour it. The pinning proxy is the obvious one. The entry
point check in validate_url_destination runs before the browser starts, and
rejected delegated names there — so the request never reached the proxy and
the suffix list had no effect. Verified against a live tunnel: without the
second change the crawl still returned "URL blocked (SSRF protection)", and
the only way through was CRAWL4AI_ALLOW_INTERNAL_URLS, which is exactly the
blunt instrument this is meant to avoid.

Deliberately an allowlist rather than a boolean, so the pin is given up only
for names an operator named, never wholesale:

  - unset (the default) leaves every path byte-identical to today;
  - a name outside the list keeps resolve-and-pin;
  - an IP literal never qualifies, because there is no name to delegate — this
    stops a suffix entry from reaching 169.254.169.254;
  - NO_PROXY still exempts hosts, matched by name since no IP is learned.

Eight tests cover that contract, four per layer: the allowlisted name reaches
the upstream unresolved, a name outside the list still arrives pinned, an IP
literal is refused with zero upstream traffic even with a wildcard suffix, and
an unset variable changes nothing.

Builds on the upstream-chaining work in unclecode#2142, which asks the upstream to
CONNECT to an already-pinned IP so the rebinding guarantee holds. That is the
right default, but it cannot serve an upstream that fronts a network whose
names do not resolve on our side, or one that enforces hostname ACLs and so
refuses CONNECT-to-an-IP — both listed there as known limitations.

CRAWL4AI_UPSTREAM_PROXY_DNS_SUFFIXES names the suffixes an operator wants the
upstream to resolve. Matching hostnames skip resolve_and_pin and are sent to
the upstream verbatim; the upstream then owns where the connection lands.

Two layers had to honour it. The pinning proxy is the obvious one. The entry
point check in validate_url_destination runs before the browser starts, and
rejected delegated names there — so the request never reached the proxy and
the suffix list had no effect. Verified against a live tunnel: without the
second change the crawl still returned "URL blocked (SSRF protection)", and
the only way through was CRAWL4AI_ALLOW_INTERNAL_URLS, which is exactly the
blunt instrument this is meant to avoid.

Deliberately an allowlist rather than a boolean, so the pin is given up only
for names an operator named, never wholesale:

  - unset (the default) leaves every path byte-identical to today;
  - a name outside the list keeps resolve-and-pin;
  - an IP literal never qualifies, because there is no name to delegate — this
    stops a suffix entry from reaching 169.254.169.254;
  - NO_PROXY still exempts hosts, matched by name since no IP is learned.

Eight tests cover that contract, four per layer: the allowlisted name reaches
the upstream unresolved, a name outside the list still arrives pinned, an IP
literal is refused with zero upstream traffic even with a wildcard suffix, and
an unset variable changes nothing.

Builds on the upstream-chaining work in unclecode#2142, which asks the upstream to
CONNECT to an already-pinned IP so the rebinding guarantee holds. That is the
right default, but it cannot serve an upstream that fronts a network whose
names do not resolve on our side, or one that enforces hostname ACLs and so
refuses CONNECT-to-an-IP — both listed there as known limitations.

CRAWL4AI_UPSTREAM_PROXY_DNS_SUFFIXES names the suffixes an operator wants the
upstream to resolve. Matching hostnames skip resolve_and_pin and are sent to
the upstream verbatim; the upstream then owns where the connection lands.

Two layers had to honour it. The pinning proxy is the obvious one. The entry
point check in validate_url_destination runs before the browser starts, and
rejected delegated names there — so the request never reached the proxy and
the suffix list had no effect. Verified against a live tunnel: without the
second change the crawl still returned "URL blocked (SSRF protection)", and
the only way through was CRAWL4AI_ALLOW_INTERNAL_URLS, which is exactly the
blunt instrument this is meant to avoid.

Deliberately an allowlist rather than a boolean, so the pin is given up only
for names an operator named, never wholesale:

  - unset (the default) leaves every path byte-identical to today;
  - a name outside the list keeps resolve-and-pin;
  - an IP literal never qualifies, because there is no name to delegate — this
    stops a suffix entry from reaching 169.254.169.254;
  - NO_PROXY still exempts hosts, matched by name since no IP is learned.

Eight tests cover that contract, four per layer: the allowlisted name reaches
the upstream unresolved, a name outside the list still arrives pinned, an IP
literal is refused with zero upstream traffic even with a wildcard suffix, and
an unset variable changes nothing.

Builds on the upstream-chaining work in unclecode#2142, which asks the upstream to
CONNECT to an already-pinned IP so the rebinding guarantee holds. That is the
right default, but it cannot serve an upstream that fronts a network whose
names do not resolve on our side, or one that enforces hostname ACLs and so
refuses CONNECT-to-an-IP — both listed there as known limitations.

CRAWL4AI_UPSTREAM_PROXY_DNS_SUFFIXES names the suffixes an operator wants the
upstream to resolve. Matching hostnames skip resolve_and_pin and are sent to
the upstream verbatim; the upstream then owns where the connection lands.

Two layers had to honour it. The pinning proxy is the obvious one. The entry
point check in validate_url_destination runs before the browser starts, and
rejected delegated names there — so the request never reached the proxy and
the suffix list had no effect. Verified against a live tunnel: without the
second change the crawl still returned "URL blocked (SSRF protection)", and
the only way through was CRAWL4AI_ALLOW_INTERNAL_URLS, which is exactly the
blunt instrument this is meant to avoid.

Deliberately an allowlist rather than a boolean, so the pin is given up only
for names an operator named, never wholesale:

  - unset (the default) leaves every path byte-identical to today;
  - a name outside the list keeps resolve-and-pin;
  - an IP literal never qualifies, because there is no name to delegate — this
    stops a suffix entry from reaching 169.254.169.254;
  - NO_PROXY still exempts hosts, matched by name since no IP is learned.

Eight tests cover that contract, four per layer: the allowlisted name reaches
the upstream unresolved, a name outside the list still arrives pinned, an IP
literal is refused with zero upstream traffic even with a wildcard suffix, and
an unset variable changes nothing.

Builds on the upstream-chaining work in unclecode#2142, which asks the upstream to
CONNECT to an already-pinned IP so the rebinding guarantee holds. That is the
right default, but it cannot serve an upstream that fronts a network whose
names do not resolve on our side, or one that enforces hostname ACLs and so
refuses CONNECT-to-an-IP — both listed there as known limitations.

CRAWL4AI_UPSTREAM_PROXY_DNS_SUFFIXES names the suffixes an operator wants the
upstream to resolve. Matching hostnames skip resolve_and_pin and are sent to
the upstream unresolved; the upstream then owns where the connection lands.

Two layers had to honour it. The pinning proxy is the obvious one. The entry
point check in validate_url_destination runs before the browser starts, and
rejected delegated names there — so the request never reached the proxy and
the suffix list had no effect. Verified against a live tunnel: without the
second change the crawl still returned "URL blocked (SSRF protection)", and
the only way through was CRAWL4AI_ALLOW_INTERNAL_URLS, which is exactly the
blunt instrument this is meant to avoid. Both layers now share one set of
helpers rather than parsing the list twice, so they cannot drift into
disagreeing about which names are delegated.

Deliberately an allowlist rather than a boolean, so the pin is given up only
for names an operator named, never wholesale:

  - unset (the default) leaves every path byte-identical to today;
  - a name outside the list keeps resolve-and-pin;
  - an IP literal never qualifies, because there is no name to delegate — this
    stops a suffix entry from reaching 169.254.169.254;
  - a wildcard entry is refused with a warning rather than honoured, since
    delegating every name would turn any caller-supplied URL into a lookup
    performed by the upstream;
  - delegated names are reachable on ports 80 and 443 only, configurable via
    CRAWL4AI_UPSTREAM_PROXY_DNS_PORTS. The pinned path leaves the port open but
    requires a global address; passthrough gives up exactly that check, so
    without a port policy a listed suffix could reach :9200 or :5432 rather
    than a web server;
  - NO_PROXY still exempts hosts, matched by name since no IP is learned.

Names are compared in normalised form — case-folded, root dot stripped,
IDNA-encoded, length checked before and after encoding — and that normalised
name is what is sent to the upstream, so the name the allowlist authorised is
the name that goes on the wire rather than the caller's original spelling.

Fifty-four tests cover the contract, including what the upstream actually
receives: a listed name reaches it unresolved while a name outside the list
still arrives pinned, three spellings of one name all arrive as the single
authorised form, an IP literal is refused with zero upstream traffic, ports
outside the allowed set fall back to the pin, a set-but-unusable port list
closes rather than opens, and an unset variable changes nothing. The delegated
plain-HTTP path carries upstream auth and Connection: close, so a reused client
connection cannot smuggle a second, unchecked request upstream.

Builds on the upstream-chaining work in unclecode#2142, which asks the upstream to
CONNECT to an already-pinned IP so the rebinding guarantee holds. That is the
right default, but it cannot serve an upstream that fronts a network whose
names do not resolve on our side, or one that enforces hostname ACLs and so
refuses CONNECT-to-an-IP — both listed there as known limitations.

CRAWL4AI_UPSTREAM_PROXY_DNS_SUFFIXES names the suffixes an operator wants the
upstream to resolve. Matching hostnames skip resolve_and_pin and are sent to
the upstream unresolved; the upstream then owns where the connection lands.

Two layers had to honour it. The pinning proxy is the obvious one. The entry
point check in validate_url_destination runs before the browser starts, and
rejected delegated names there — so the request never reached the proxy and
the suffix list had no effect. Verified against a live tunnel: without the
second change the crawl still returned "URL blocked (SSRF protection)", and
the only way through was CRAWL4AI_ALLOW_INTERNAL_URLS, which is exactly the
blunt instrument this is meant to avoid. Both layers now share one set of
helpers rather than parsing the list twice, so they cannot drift into
disagreeing about which names are delegated.

Deliberately an allowlist rather than a boolean, so the pin is given up only
for names an operator named, never wholesale:

  - unset (the default) leaves every path byte-identical to today;
  - a name outside the list keeps resolve-and-pin;
  - an IP literal never qualifies, because there is no name to delegate — this
    stops a suffix entry from reaching 169.254.169.254;
  - a wildcard entry is refused with a warning rather than honoured, since
    delegating every name would turn any caller-supplied URL into a lookup
    performed by the upstream;
  - delegated names are reachable on ports 80 and 443 only, configurable via
    CRAWL4AI_UPSTREAM_PROXY_DNS_PORTS. The pinned path leaves the port open but
    requires a global address; passthrough gives up exactly that check, so
    without a port policy a listed suffix could reach :9200 or :5432 rather
    than a web server;
  - NO_PROXY still exempts hosts, matched by name since no IP is learned.

Names are compared in normalised form — case-folded, root dot stripped,
IDNA-encoded, length checked before and after encoding — and that normalised
name is what is sent to the upstream, so the name the allowlist authorised is
the name that goes on the wire rather than the caller's original spelling.

Fifty-four tests cover the contract, including what the upstream actually
receives: a listed name reaches it unresolved while a name outside the list
still arrives pinned, three spellings of one name all arrive as the single
authorised form, an IP literal is refused with zero upstream traffic, ports
outside the allowed set fall back to the pin, a set-but-unusable port list
closes rather than opens, and an unset variable changes nothing. The delegated
plain-HTTP path carries upstream auth and Connection: close, so a reused client
connection cannot smuggle a second, unchecked request upstream.

CRAWL4AI_UPSTREAM_PROXY_ONLY_SUFFIXES narrows which destinations are chained at
all. Chaining is otherwise all-or-nothing — with an upstream set every target
goes through it, and NO_PROXY only subtracts exceptions from that — so a single
deployment cannot serve an internal-only upstream and direct public egress at
the same time, because the public set cannot be enumerated in NO_PROXY. Sending
public crawls through the upstream anyway is not just wasteful: an upstream that
authorises per destination refuses them, and the fetch is attributed to its
network rather than ours. Listing suffixes here inverts the rule for those names
— they are chained, everything else dials direct.

It composes with CRAWL4AI_UPSTREAM_PROXY_DNS_SUFFIXES rather than overriding it:
a name may be handed over unresolved only if it is also allowed to use the
upstream, so neither list can widen what the other permits. Unset, the default,
leaves every path byte-identical to today. The change only ever adds a reason to
skip the upstream, so a target that is not chained is resolved and pinned exactly
as before — the pin covers strictly more traffic than it did, never less.

Six further tests cover it: an unset variable changes nothing, a listed suffix is
still chained as the pinned IP, an unlisted host produces zero upstream traffic,
a substring such as notcorp.example does not match .corp.example, delegation is
refused when the DNS list allows a name the upstream allowlist does not, and
wildcard or malformed entries are ignored with the list then behaving as absent.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Builds on the upstream-chaining work in unclecode#2142, which asks the upstream to
CONNECT to an already-pinned IP so the rebinding guarantee holds. That is the
right default, but it cannot serve an upstream that fronts a network whose
names do not resolve on our side, or one that enforces hostname ACLs and so
refuses CONNECT-to-an-IP — both listed there as known limitations.

CRAWL4AI_UPSTREAM_PROXY_DNS_SUFFIXES names the suffixes an operator wants the
upstream to resolve. Matching hostnames skip resolve_and_pin and are sent to
the upstream unresolved; the upstream then owns where the connection lands.

Two layers had to honour it. The pinning proxy is the obvious one. The entry
point check in validate_url_destination runs before the browser starts, and
rejected delegated names there — so the request never reached the proxy and
the suffix list had no effect. Verified against a live tunnel: without the
second change the crawl still returned "URL blocked (SSRF protection)", and
the only way through was CRAWL4AI_ALLOW_INTERNAL_URLS, which is exactly the
blunt instrument this is meant to avoid. Both layers now share one set of
helpers rather than parsing the list twice, so they cannot drift into
disagreeing about which names are delegated.

Deliberately an allowlist rather than a boolean, so the pin is given up only
for names an operator named, never wholesale:

  - unset (the default) leaves every path byte-identical to today;
  - a name outside the list keeps resolve-and-pin;
  - an IP literal never qualifies, because there is no name to delegate — this
    stops a suffix entry from reaching 169.254.169.254;
  - a wildcard entry is refused with a warning rather than honoured, since
    delegating every name would turn any caller-supplied URL into a lookup
    performed by the upstream;
  - delegated names are reachable on ports 80 and 443 only, configurable via
    CRAWL4AI_UPSTREAM_PROXY_DNS_PORTS. The pinned path leaves the port open but
    requires a global address; passthrough gives up exactly that check, so
    without a port policy a listed suffix could reach :9200 or :5432 rather
    than a web server;
  - NO_PROXY still exempts hosts, matched by name since no IP is learned.

Names are compared in normalised form — case-folded, root dot stripped,
IDNA-encoded, length checked before and after encoding — and that normalised
name is what is sent to the upstream, so the name the allowlist authorised is
the name that goes on the wire rather than the caller's original spelling.

Fifty-four tests cover the contract, including what the upstream actually
receives: a listed name reaches it unresolved while a name outside the list
still arrives pinned, three spellings of one name all arrive as the single
authorised form, an IP literal is refused with zero upstream traffic, ports
outside the allowed set fall back to the pin, a set-but-unusable port list
closes rather than opens, and an unset variable changes nothing. The delegated
plain-HTTP path carries upstream auth and Connection: close, so a reused client
connection cannot smuggle a second, unchecked request upstream.

CRAWL4AI_UPSTREAM_PROXY_ONLY_SUFFIXES narrows which destinations are chained at
all. Chaining is otherwise all-or-nothing — with an upstream set every target
goes through it, and NO_PROXY only subtracts exceptions from that — so a single
deployment cannot serve an internal-only upstream and direct public egress at
the same time, because the public set cannot be enumerated in NO_PROXY. Sending
public crawls through the upstream anyway is not just wasteful: an upstream that
authorises per destination refuses them, and the fetch is attributed to its
network rather than ours. Listing suffixes here inverts the rule for those names
— they are chained, everything else dials direct.

It composes with CRAWL4AI_UPSTREAM_PROXY_DNS_SUFFIXES rather than overriding it:
a name may be handed over unresolved only if it is also allowed to use the
upstream, so neither list can widen what the other permits. Unset, the default,
leaves every path byte-identical to today. The change only ever adds a reason to
skip the upstream, so a target that is not chained is resolved and pinned exactly
as before — the pin covers strictly more traffic than it did, never less.

Six further tests cover it: an unset variable changes nothing, a listed suffix is
still chained as the pinned IP, an unlisted host produces zero upstream traffic,
a substring such as notcorp.example does not match .corp.example, delegation is
refused when the DNS list allows a name the upstream allowlist does not, and
wildcard or malformed entries are ignored with the list then behaving as absent.